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DETAILED ACTION 

Specification 

1 . The disclosure is objected to because of the following informalities: In page 3, 
lines 14-16, the applicant makes use of quotation marks in providing a description of the 
behavior of the system disclosed. It is unclear to the examiner what the purpose of 
these marks is, since the text does not seem to contain a special use of language, nor is 
it a quotation of some external source. If the text does in fact refer to a special meaning 
of the language used, then further explanations are required. If the text uses language 
directly extracted from an external source, then that source should be referenced. If 
neither of these possibilities applies, and there is no other objective to the quotation 
marks, the applicant should remove them. 

Appropriate correction is required. 

2. The abstract of the disclosure is objected to because it is written using many 
paragraphs. The abstract should be only one paragraph. Correction is required. See 
MPEP§ 608.01(b). 

Claim Objections 

3. Claims 5, 9-1 1 are objected to because of the following informalities: These 
claims make use of quotation marks. It is unclear to the examiner what the purpose of 
these marks is, since the text does not seem to contain a special use of language, nor is 
it a quotation of some external source. Even in the case that the text is a direct 
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quotation, such as listing a limitation from a parent claim, the use of quotation marks is 
unnecessary. Appropriate correction is required. 

4. Claim 21 is objected to because of the following informalities: Claim 21 makes 
reference to the "methods" disclosed in claim 1 . There is only one method disclosed in 
claim 1. Appropriate correction is required. 



Claim Rejections - 35 USC § 101 

5. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

6. Claims 19 and 20 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. Both claims 19 and 20 are directed 
to computer programs per se, which constitute non-statutory subject matter. The claims 
scope includes the mere code listings of said computer programs. 

7. Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. Claim 21 is directed to data per se which 
constitutes non-statutory matter. 

8. Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. Claim 22 is directed to a storage medium that 
contains data which does not imply any functionality. The fact that this data is 
embedded in a computer readable medium does not render the claim patentable, since 
such a result would exalt form over substance. Furthermore, the specification definition 
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of a storage medium, at page 17, lines 21 and 22, fails to limit the embodiments for this 
storage medium. Therefore, the claim is directed to embodiments that constitute non- 
statutory subject matter for this storage medium, such as carrier waves. 

Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

10. Claims 1-12 and 14-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Allard (U.S. Patent Publication No. 2003/0216834 A1) and Aliaga et 
al. (U.S. Patent Publication No. 2002/0176635 A1, "Aliaga" hereinafter). 

Regarding claim 1 , Allard teaches an "image generating method" (i.e., a method 
of providing a user interface, see par. [0013], with a heads-up display, see par. [0044]) 
"comprising the following steps: (1 ) a step of receiving environment information acquired 
by one or a plurality of space measurement sensor attached to a moving body" (i.e., 
receiving sensor information, see par. [0035], from sensors mounted on a robot, see 
par. [0036], where the robot can move, see par. [0034]); "(2) a step of receiving time 
when the environment information is received, and parameter of the space 
measurement sensor itself at the time" (implicitly, since the display can vary the 
representation of the data gathered based on the age of the data, see par. [0050], and, 
since one of the possible sensors is a spinning sonar scanner, see par. [0036], at least 
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the angle towards which the sonar is pointed to must be known); "(3) a step of saving 
history information representing the environment information, the time, and the 
parameter" (see par. [0050] for using the age of the sensor data in a map representing 
the environment of the robot, as well as representing a history of past positions of the 
robot); "(....) and (5) a step of generating virtual environment image seen from the (....) 
observation point based on the saved history information" (see par. [0053] for creating a 
panorama image from a series of snapshots, as well as storing individual snapshots). 

Allard does not teach "(4) a step of receiving designation for virtual observation 
point" or that the virtual image is created with respect to this virtual observation point. 
Aliaga, however, teaches a method of creating new images from stitching together data 
from captured image sequences (see par. [0014]), and allowing a user to select a virtual 
point of view from which the virtual image is created (see par. [0015]). Because both 
Allard and Aliaga use a method of stitching pictures together in order to obtain a virtual 
panoramic image, as discussed above, and since Allard discusses the advantage of 
having the camera outside of the robot in order to obtain perspectives unattainable from 
the robot, it would have been obvious to one of ordinary skill in the art at the time of the 
invention to combine the virtual panoramic image creation from the robot as disclosed in 
Allard, with the virtual image generation from a user selectable virtual viewing point, as 
disclosed in Aliaga, in order to provide an advantageous perspective of the 
environment. 
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Regarding claim 2, Allard further discloses "(6) a step of generating image of the 
moving body itself seen from the (....) observation point based parameter of the moving 
body itself (see par. [0065] for overlaying an image representing the robot within the 
display of the environment image, representing positions of the robot). Allard does not 
disclose that the viewing point is a virtual point, but that limitation has been addressed 
for claim 1 above. Allard also discloses "(7) a step of generating a composite image 
including the image of the moving body itself and the virtual environment image, using 
the virtual environment image and the image of the moving body itself (implicitly, since 
the image of the robot is overlayed on the environment image). 

Regarding claim 3, Allard further teaches that "the environment image is a 
plurality of still pictures" (see par. [0053] for generating a panoramic image from a series 
of still images). 

Regarding claim 4, Allard further teaches that "the environment image is moving 
picture" (see par. [0039] for streaming MJPEG, Motion JPEG, from the robot). 

Regarding claim 5, Allard further discloses that "the parameter of the moving 
body itself in step (6) is for any time point between a time point when a virtual 
observation point is designated, or close to that time point, to a time point when a 
generated composite image is presented" (see par. [0058] for representing the pose of 
the robot at a given moment). 

Regarding claim 6, Allard further teaches that "the moving body can propel itself 
(see par. [0034] for a description of several means for the robot to move). 
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Regarding claim 7, Aliaga also discloses that "the virtual observation point exists 
at a position looking at the environment around the moving body and/or the environment 
around a point the operator wants to see" (see par. [0015] for allowing a user to select 
the position of the virtual viewing point). 

Regarding claim 8, Aliaga further discloses that "the virtual observation point 
exists at a position looking at the moving body from behind" (since the user can select 
to place the virtual viewing point at any position). 

Regarding claim 9, Allard also teaches that "the parameter of the space 
measurement sensor itself in step (2) includes position and attitude of space 
measurement sensor itself and/or data, matrix or table representing a relationship 
between data space acquired by the space sensor itself and real space" (implicitly, 
since the sensor can be a spinning sonar scanner, see par. [0036], which inherently 
requires knowledge of the relative angle of the sonar, and provides range information). 

Regarding claim 10, Aliaga further teaches that "generating based on history 
information in step (5) is selection of any image contained in the environmental 
information based on closeness of position of the space measurement sensor itself at 
the time the environmental information is acquired, and the virtual observation point" 
(see par. [0030] for storing camera position and attitude as images are taken, see par. 
[0038] for using the images taken closest to the virtual viewing point). 

Regarding claim 1 1 , Aliaga further teaches that "the generation based on history 
information in step (5) is new generation based on history information" (i.e., the image 
generation is a dynamic process using static images, see par. [0015]). 
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Regarding claim 12, Allard also discloses that "the virtual environment image is 
still picture" (see par. [0052] for generation of panoramic pictures, and see par. [0053], 
for requiring the user to update the panoramic image). 

Regarding claim 14, Allard also teaches that "position of the moving body is 
included in the parameter of the moving body itself (see par. [0076] for storing the 
position of the robot as obtained from a GPS sensor). 

Regarding claim 15, Allard also teaches that "attitude of the moving body is 
further included in the parameter of the moving body itself (see par. [0058] for 
presenting the current pose of the robot). 

Regarding claim 16, the combination of Allard and Aliaga, as discussed for 
claims 1 and 2, teaches a "presentation method for presenting a composite image 
generated using the method of claim 2" (since the methods disclosed in Allard and 
Aliaga both are for presenting information to a user). 

Regarding claim 17, Allard discloses an "image generating system" (a system 
generating a user interface to control a robot, see par. [0013], see par. [0033] to [0041] 
for hardware components), "comprising a moving body" (i.e., a robot, see par. [0034]), 
"(....) the moving body being provided with space measurement sensor for acquiring 
environment information" (see par. [0035] and [0036] for sensors mounted in the robot 
to obtain environment information), "a control section (....) wherein the control section 
carries out the following functions: (a) a function of saving history information 
representing the environmental information" (see par. [0050] for using past sensor 
information, as well as the age of the information), "the time when the environmental 
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information was acquired" (see par. [0050], where the age of the information is used), 
"and parameter of the space measurement sensor it self at the time the environmental 
information was acquired" (implicitly, since a possible sensor is a spinning sonar 
scanner, which requires knowledge of the relative angle of the sensor at the time of 
measurement) "(....) and an information acquisition section (....)" (see par. [0038] for 
receiving information from the robot). 

Allard does not disclose "(b) a function of receiving information for designated 
virtual observation point; and (c) a function of generating virtual environment image 
seen from the virtual observation point based on the saved history information". Aliaga, 
however, teaches a method of creating new images from stitching together data from 
captured image sequences (see par. [0014]), and allowing a user to select a virtual 
point of view from which the virtual image is created (see par. [0015]), performed by a 
computer, see par. [0053]. Because both Allard and Aliaga disclose systems for 
stitching pictures together in order to obtain a virtual panoramic image (see Allard, par. 
[0053]), as discussed above, and since Allard discusses the advantage of having the 
camera outside of the robot in order to obtain perspectives unattainable from the robot 
(see par. [0096]), it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine the virtual panoramic image creation from the robot as 
disclosed in Allard, with the virtual image generation from a user selectable virtual 
viewing point, as disclosed in Aliaga, in order to provide an advantageous perspective 
of the environment. 
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Regarding claim 18, Allard teaches "further comprising an information acquisition 
section, the information acquisition section being for acquiring parameters of the moving 
body itself (see par. [0076] for receiving positions for the robot), "wherein the control 
section further carries out the following functions: (d) a function of generating image of 
the moving body itself seen from the (....) observation point based on parameter of the 
moving body itself (see par. [0065] for overlaying an image representing the robot 
within the display of the environment image, representing positions of the robot). Allard 
does not disclose that the viewing point is a virtual point, but that limitation has been 
addressed for claim 17 above; Allard also discloses "(e) a function of generating a 
composite image including an image of the moving body and the virtual environment 
image, using the virtual environment image and the image of the moving body itself 
(implicitly, since the image of the robot is overlayed on the environment image). 

Regarding claim 19, the combination of Allard and Aliaga, as discussed for claim 
1 above, teaches a "computer program for causing a computer to execute the steps of 
the methods of claim 1" (since the methods disclosed are to be executed by computers, 
see Allard, par. [0040], and Aliaga, par. [0053]). 

Regarding claim 20, the combination of Allard and Aliaga, as discussed for claim 
17 above, teaches a "computer program for causing a computer to execute the function 
of the control section of claim 17" (since the methods disclosed are to be executed by 
computers, see Allard, par. [0040], and Aliaga, par. [0053]). 
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Regarding claim 21, the combination of Allard and Aliaga, as discussed for claim 
1 above, teaches "data containing information representing the virtual environment 
image or the composite image generated using the generating methods of claim 1" 
(since a digital representation of both virtual image and data from real environment are 
generated, see discussion for claim 1 above). 

Regarding claim 22, the combination of Allard and Aliaga, as discussed for claim 
21 above, teaches a "storage medium for storing the data of claim 21" (since the 
methods disclosed are to be executed by computers, see Allard, par. [0040], and 
Aliaga, par. [0053], they inherently require a computer-readable storage medium 
containing the data in order to operate on said data). 

11. Claim 13 is rejected under 35 U.S. C. 103(a) as being unpatentable over Allard 
(U.S. Patent Publication No. 2003/0216834 A1) and Aliaga et al. (U.S. Patent 
Publication No. 2002/0176635 A1 , "Aliaga" hereinafter) as applied to claim 2 above, and 
further in view of Hiroyuki et al. (Japanese Patent Publication No. 2000-237451 , 
provided by the applicant, "Hiroyuki", hereinafter). 

Regarding claim 13, neither Allard nor Aliaga teach that "the image of the moving 
body itself contained in the composite image of step (7) is a semi-transparent image, a 
transparent image, or a wireframe image". Hiroyuki, however, teaches a method of 
displaying a vehicle overlayed over a background, using transparency, in order to allow 
better visibility for a user who is controlling a vehicle in the simulation, see abstract. 
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Because both Allard and Hiroyuki disclose methods for improving the visibility of an 
environment in which a moving object is controlled (see Allard, par. [0096]), it would 
have been obvious to combine the moving object representation shown in Allard with 
the transparency technique shown in Hiroyuki in order to allow a user to see objects 
obstructed by the representation of the moving object. Moreover, since both Hiroyuki 
and Aliaga teach methods in which a background image simulating an environment is 
generated and modified based on the user actions representing a movement of the 
focal point (see Aliaga, par. [0015], and see Fig. 7 in Hiroyuki), it would have been 
obvious to one of ordinary skill in the art to combine the transparency technique taught 
by the combination of Hiroyuki and Allard, with the virtual image and user selectable 
viewpoint as disclosed in Aliaga, in order to provide a semi-transparent image of a 
moving object overlayed on a generated panoramic scene, facilitating the control of said 
moving object. 

Conclusion 

1 2. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to CARLOS PERROMAT whose telephone number is 
(571) 270-7174. The examiner can normally be reached on M-TH 8:30 am- 5:00 pm 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kieu-Oanh Bui can be reached on (571) 272-7291 . The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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